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ASSOCIATING DATA WITH IMAGES IN IMAGING SYSTEMS 

Related Application Data 
This patent application is a continuation in part of co-pending US Patent Application 

No. (Attorney Docket 601 12), entitled WATERMARK EMBEDDER AND 

READER filed February 14, 2000 by Rhoads et aL, which is hereby incorporated by 
reference. 

Field of the Invention 
The invention relates to still and moving image capture devices, and more 
particularly relates to associating auxiliary data with images captured in such devices. 



Background and Summary of the Invention 
Prior art still image and movie cameras memorialized image data on media (e.g., 
film or magnetic tape), but did not include provision to store other useful information. 

Some years ago 5 still-image cameras with the capability to insert visible date text in a 
1 5 corner of the image frame became available, solving the problem of trying to recall the dates 
that photos of interest were taken. 

More recently, the Advanced Photo System provided a magnetic encoding medium 
on the back of photographic film, permitting additional information to be stored (e.g., 
exposure information, whether flash was used, desired print size, etc.). 
20 With the growing popularity of digital storage of image data, it would be helpful if 

auxiliary data could be stored in association with a photo. Such storage is now possible in 
the header fields of certain popular data formats, but such data can be lost if the file is 
converted to another format. 

In one embodiment of the invention, auxiliary data about an image is 
25 steganographically encoded ("watermarked") within the image itself. By integrating the 
image and the associated data in this fashion, the auxiliary data cannot become separated 
from the image. A great number of useful systems are thereby reliably enabled. 

In another embodiment, a reference to auxiliary data is steganographically encoded 
within the image. The auxiliary data is stored in a metadata database that maps the reference 
30 encoded in an image to its corresponding metadata. Various imaging application programs 
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and devices can access the metadata by extracting the reference from an image, and 
querying the database for metadata corresponding to the reference. The metadata database 
may be implemented in a network server to make metadata readily available to devices and 
applications distributed throughout a computer network. 
5 The steganographic link may be used to retrieve image metadata across imaging 

systems. In one configuration, for example, imaging systems have metadata servers that 
maintain metadata for a collection of images. The metadata server is responsible for 
responding to requests for metadata associated with images in its collection. In the event 
that a metadata server does not have metadata associated with a particular image, it forwards 

1 0 the reference extracted from the image to a metadata router, which in turn, re-directs the 
request to the appropriate metadata server. 

The steganographic data may be embedded in an image within an image capture 
device, or elsewhere. In one embodiment, the steganographic data is embedded in the image 
as part of the process of uploading it from the camera. In another embodiment, an image 

1 5 capture device embeds steganographic data in the image before uploading it to an external 
device. 

The foregoing and additional features and advantages will be more readily apparent 
from the following detailed description, which proceeds with reference to the accompanying 
drawings. 

20 

Brief Description of the Drawings 
Fig. 1 is a block diagram illustrating an example of a digital camera. 
Fig. 2 is a block diagram illustrating various connectivity configurations for 
transferring data to and from an image capture device. 
25 Fig. 3 is a diagram illustrating an application of steganographic links to metadata in 

images. 

Fig. 4 is a diagram illustrating the use of steganographic links to access metadata 
from different imaging systems. 
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Detailed Description 

Introduction 

As noted above, there are many applications for associating auxiliary data with 
images captured in an image capture device. A short list of these applications include: 
5 • managing copyrights, 

• authenticating images, 

• describing objects within an image, 

• describing the context of the image (time, place, subject, camera properties), 

• facilitating image searching based on content, 

10 • uploading and organizing images from cameras in photo library databases, 

• sharing images over networks, such as the Internet 

• linking derivative images to originals. 

The following sections describe various implementations of image capture devices 
and related systems that associate various forms of data with images. The sections discuss 

15 associating data in images using steganographic and other techniques. Steganographic 

methods can encode data in an image, including identifiers and references that associate the 
image with additional data stored outside the image itself. In some applications, the 
capacity of the images to hold a steganographic signal, such as an imperceptible watermark, 
is sufficient to store the data associated with the image. In other applications, it is useful to 

20 store additional data elsewhere, and refer to it through a reference hidden in the image. 

One way to associate data with an image is to store the data in the image container 
(e.g., a file), but outside the image. Yet another way is store the data in a location external 
to the image file, and create an association between the image file and the external data. In 
this case, the image may be associated with the external data via a reference encoded in the 

25 image itself or in the image file that refers to the external data. Data associated with a 
digital object, such as an image, is sometimes referred to as "metadata." 

In some applications, the image capture device enables users to specify the types of 
data as well as specific items of data to be associated with images captured in it. The data 
"type" describes what a data item represents, while the data "item" refers to the actual data 

30 associated with the image. Some examples of data types include the time, place, and subject 
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of the image, and examples of corresponding data items include: 12:30 PM, Times Square, 
Mom (corresponding to datatypes of time, place and subject). 

The user may specify the data types through the user interface of the image capture 
device, or through the user interface of an external device or system, which in turn, 
5 communicates the desired data types to the image capture device. Camera user interface 
components, such as buttons or voice recognition modules, enable the user to change 
settings of the device anywhere, without the need to link it to external device. Data entry 
and device configuration through external devices, such as a personal computer, personal 
digital assistant or phone, provides a number of advantages, including remote control of the 

10 camera, access to more user friendly user interfaces, and more powerful data manipulation 
and storage than is capable on the camera. 

In addition to specifying the data type, the user may also specify actual data items to 
be associated with images. Again, the user may specify the data items through the user 
interface of the image capture device, or through the user interface of an external device. In 

15 cases where the user supplies data items to the external device, these items may be 

downloaded to a storage device on the image capture device, associated with images at the 
time of or after image capture, and later transferred from the device along with the 
associated images to external storage. Alternatively, the data items may be kept in storage 
external to the image capture device and associated with selected images by reference, such 

20 as through an identifier that matches an identifier hidden in the image. For example, the 
camera or some other image management system can insert a stegranographic link (e.g., in 
an image watermark) that associates the image with the desired data items. 

Image Capture Devices and Connectivity to External Devices and Systems 

25 The methods and systems for associating data with images can be implemented in 

many different types of image capture devices, including conventional film cameras, digital 
cameras and scanners. To simplify the discussion, the following description uses digital 
cameras as an example of one type of image capture device. 
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Digital Cameras 

The design of digital cameras is well understood to those skilled in the art, so is not 
belabored here. Fig. 1 shows an example of a camera 10, including its user interface 12 and 
its internal architecture. 
5 The camera 10 includes an optical system 14, an image sensor 16, and an image 

formatter 18. The optical system contains the camera's lens, focus control, and other optics 
components that control the transfer of light to the camera's sensor 16. The sensor is 
comprised of an array of discrete sensor elements arranged in clusters sensitive to three 
primary colors (e.g., red, green, blue). Two common types of sensors are CCD and CMOS 
10 sensors. The image formatter includes an analog to digital converter to convert signals from 
the sensor elements to digital form. It also includes a color space converter to map the 
signal into a desired color space, such as RGB or YUV. The formatter transforms the image 
signal into a form suitable for further processing and storage and stores it in the camera's 
memory subsystem. 

15 Because the specific interconnect varies with the implementation, Fig. 1 depicts a 

general interconnect 22 that represents the data and control path among various components 
in the camera. In addition to the image capture components described above, the camera 
includes a processing unit 22, a memory subsystem 20, and various I/O devices. The 
camera may include one or more processing devices, such as a microprocessor, and a DSP. 

20 The processing unit executes an operating system, such as Vx Works real time 

embedded operating system in the Digita Operating Environment from Flashpoint, Inc., or 
Windows CE from Microsoft Corporation. Application programs supporting functions 
described throughout this document may be developed using software development 
environments designed for these and other commercially available operating systems, such 

25 as Tornado Tools from Wind River. 

A steganographic embedder may be implemented as an application program that 
executes on the processing unit, or in special purpose hardware that has access to the 
memory subsystem. 

The memory subsystem 20 includes a combination of ROM, RAM, and removable 
30 storage devices such as a flash memory card. 
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The I/O devices include a display 24, and input controls 26. The upper portion of 
Fig. 1 depicts an example of the rear panel of the camera showing user interface components 
on the camera. The panel has buttons and switches 30, 32, 34 that control operating modes 
of the camera and display modes on the display screen 36. It also includes scroll buttons 38- 
5 44 that enable the user to scroll through options on various display screens. 

The camera is equipped with one or more external interfaces to transfer images, 
image related data, and operating parameters to and from external devices. The type and 
number of interfaces for external connectivity vary from camera to camera, and include: 

• ports 50 for wire connections to external devices such as serial ports, USB ports, 
1 0 parallel ports, PCI, Firewire or Ilink ports (IEEE 1 394 and USB2 ports), and 

• transceivers and receivers 52 for wireless connections such as an infrared 
transceiver, RF transceivers, FM receivers, etc. 

The camera may also include a cellular or conventional modem 54 for transferring 
data to and from a telephone network. In addition to transferring images and data via 
1 5 connections to external devices, the camera can also receive and send data via a removable 
memory device. 

Connectivity to External Devices 

The camera's connectivity features enable it to upload images and data to external 
devices, and enables external devices to download operating parameters and image related 
20 data (e.g., image metadata). There are numerous connectivity configurations, including 
wired and wireless connections to stand-alone and networked devices. 

Fig. 2 is diagram illustrating some examples of configurations for connecting the 
camera with external devices. As shown, the camera 100 may connect directly to a 
communication network 102, like the Internet, it may connect to a networked device such as 
25 a personal computer or telephone (104), and finally, it may connect to a stand-alone device 
106. 

In order to connect to a network, the camera may have a cellular or conventional 
modem or a network adapter. It may be connected to a networked or stand-alone device via 
a communication interface such as the ones mentioned in the previous section. 



V 
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Fig. 2 highlights some of the connectivity scenarios made possible by these 
connectivity components in the camera. The connectivity scenarios can be grouped into two 
principle categories: 1) transferring data and operating parameters to the camera; and 2) 
uploading images, data associated with the images, and operating parameters from the 
5 camera. 

Transfers to the Camera 

One configuration is to transfer the desired information through a direct connection 
to the camera, such as through the camera's serial port, USB port, parallel port, Firewire or 
Ilink port, infrared transceiver port, or RF transceivers port. In this scenario, the source of 

10 the information may be a networked or stand-alone device. 

Another configuration is through an indirect connection from the source of the 
information to the camera. The indirect connection may include one or more hops through a 
wired or wireless connection. An example of a wired connection is where the camera 100 is 
connected to a network 102, such as the Internet, and another device, such as a server 108, 

1 5 sends the information through the network to the camera, which is connected to the network. 
A useful application of this configuration is where a user of a client computer 110 designates 
operating parameters and metadata through an interactive web site. The web server 108, in 
turn, transfers the operating parameters and metadata to the camera through the network 
102. 

20 Another related configuration is where a user specifies desired parameters to a server 

computer 112, which in turn, sends the data to the camera through a wireless connection or 
broadcast. One example of a wireless connection is through wireless network to a cellular 
modem in the camera. Example of a wireless broadcast include transmitting the data from 
the server to a FM transmitter (e.g., the radio tower 1 14) or satellite transmitter (116, 1 18) to 

25 a compatible receiver in the camera. 

Uploading Data from the Camera 

Many of the same configurations outlined above for transferring data to the camera 
apply to uploading data from the camera as well. However, as practical matter, cameras 
have more limited resources to process and transmit data. One way to address these 
30 limitations is to incorporate into the camera a high bandwidth interface to transfer large 
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amounts of data, such as the images and their related metadata, as well as a low bandwidth 
interface for transferring smaller packages of data, such as an identifier or set of control 
signals. The high bandwidth interface may be implemented using a port that can 
communicate large amounts of data efficiently, without unduly complicating the circuitry on 
the camera. Examples of such ports include a USB, firewire or Ilink port (IEEE 1394 and 
USB2). The low bandwidth interface may be used to implement a wireless signal 
transmitter, such as cellular modem transceiver, or other wireless signal transmitter, such as 
FM, RF, infrared, etc. 

Image Data Configuration 

As noted above, there are two principle categories of methods for configuring the 
camera to specify its operating parameters, and to specify the types of auxiliary data to be 
associated with images: 1) through the user interface of the camera; and 2) through an 
external device. The following sections provide examples of devices for configuring the 
camera and methods of operating these devices. 

Camera UI 

The camera UI may be implemented with many different combinations of input and 
output devices. The input devices may be implemented using combinations of hardware 
controls, such as buttons, switches, dials, cursor control devices, etc., and soft controls, such 
as touch screens, soft buttons, scroll bars, and check boxes displayed on a display device. 
The camera may also be configured to receive voice input through a microphone, voice 
codec, and voice recognition hardware and software. 

Similarly, the output devices may produce visual and/or audio output. For example, 
the camera may have a display screen, with or without support for video output. In addition, 
it may have audio output, and allow for playback of voice instructions, and output of a text 
to speech synthesis system. 

In the example depicted in Fig. 1, the user manipulates the user interface (Fig. 1) to 
specify the data that is to be associated with a picture or series of pictures. The user may 
specify the data type to be associated with a picture, and then allow the camera to supply the 
actual data from its storage or from an internal device (e.g., clock for the time, GPS for the 
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location, etc.). The data in the storage device may, for example, be downloaded to the 
camera from an external source. Additionally, the user may specify the actual data to be 
associated with each picture, either in advance, or as he snaps the pictures. For example, the 
photographer might enter his name, and the names of the people or things in the picture. 
The photographer may opt to enter the data once, and then have it linked to several pictures 
in a virtual roll. A virtual roll is a set of images that constitutes a "roll." 

In one particular implementation depicted in Fig. 1, the user interface accepts user 
input via the buttons 30-34. Using known UI techniques, the user can use the scroll buttons 
38-44) to scroll through a list of selections displayed on display screen 36 until a desired 
selection is presented. Once the list has been scrolled to the desired entry, the user can 
actuate a second button (e.g., 32-34) to effect that selection. The data types associated with 
an image or set of images can be of various classes. One class can identify the 
photographer, another class can identify the subject, etc. 

For the class of photographers, the scrollable list of selections can include a default 
list of descriptors (e.g., Mom, Dad, Childl, Child2, #1, #2, etc.), supplemented (or replaced 
if desired) by a list that is customized by the owner of the camera (e.g., Bill, Kristen, 

Hannah, David, etc.). 

The class of subjects can similarly include a default list (e.g., Birthday, Vacation, 
Anniversary, Wedding, House, Car, Pet, etc.) and/or a customized list (Uncle Harry, 
Yellowstone, Mushrooms, Seascapes, etc.) The user interface for selection of subjects may 
permit selection of several subjects - providing alternate descriptors for an image. 
A descriptor selected by the user can be used to encode the picture just-snapped, or can be 
used to encode pictures thereafter-snapped. A descriptor embedded in the image may be in 
the form of text, or a number that refers to metadata stored outside the image. 

Rather than having a cumbersome interface built into the camera, the camera may 
support one or more plug-in user interface peripherals, such as a keyboard. Alternatively, 
the configuration data in the camera may be synchronized with configuration data in an 
external device via a wire or wireless connection. 

In some applications, the photographer may be intentionally precluded from 
controlling the data associated with pictures. This feature is important in applications where 
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there is concern that the photographer might tamper with the metadata associated with an 
image. For example, the photographer might try to alter the time, place or subject to create a 
false or misleading photographic record. To prevent such tampering, the photographer's 
employer or supervisor can set the operating parameters and make data type selections from 
an external source. Operating parameters may be used to set certain types of data, such as 
the time, or to disable the photographer's ability to change certain parameters. 

In order to implement such features, the camera supports a variety of different 
operating modes. In a session mode, the camera operates under the control of parameters 
that govern that session. The operating parameters for the session may be established 
through the user interface of the camera, or from an external device. In addition, the user 
may initiate a session, or an external device may initiate the session. 

When the camera receives the operating parameters for the session, it makes all 
settings to comply with the instructions specified in the operating parameters. For example, 
if the session parameters instruct the camera to set the time, date, place or subject for the 
session, the camera does so. The session parameters may also specify that the user is to be 
precluded from altering the operating parameters for the duration of the session. For 
example, the user may be precluded from altering data items or data types associated with 
images snapped during the session or from altering certain camera functions, such as a time 
keeping function, a GPS function, etc. during the session. 

The duration of the session may be set and measured using certain operating 
parameters. For example, the session may be in effect for specified period of time, for 
selected number of images, for selected GPS locations within a defined range, etc. In 
addition, the session duration may be specified in terms of certain events, such as a 
configuration event in which the session is configured, and an upload event, where the 
images snapped during a session are uploaded from the camera. Using this approach, the 
one configuring the camera can specify that the session shall extend from the configuration 
event to some other event, such as the act of uploading the images in the session. 

The camera can be programmed to operate according to the parameters that govern 
the session. It may disallow the user from capturing images when the camera travels outside 
a particular location specified in the session parameters and monitored by the GPS unit 
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within the device. Similarly, it may disallow the user from capturing images outside a 
predetermined time interval specified in the session parameters. To this end, the camera has 
a time clock that keeps time and is synchronized to a master clock to avoid errors that might 
occur due to user tampering or traveling through different time zones. The time clock can 
5 be set permanently to run synchronized with a master clock or can be periodically 

synchronized by an external master clock via a wire or wireless connection. Just as the 
parameters can prevent certain actions outside a given time span or geographic area, they 
can also selectively allow certain actions within a given time span or area. In addition, the 
time clock and GPS unit can be used to insert or link a time and place stamp to every image 
1 0 that is captured during a session or otherwise. 

The session parameters can govern when and where the images are 
steganographically marked and uploaded. For example, the parameters may specify that the 
images are to be steganographically marked at the time of image capture, or at some later 
time. Similarly, the parameters may specify that the images are to be steganographically 
1 5 marked at the location of image capture or some other location (within some geographic 
region), as determined by the GPS unit within the device. Similar parameters may be 
established to specify a time or time range for when the image or images may be uploaded to 
an external storage device or processor. 

A session may be tracked and associated with session related metadata by a session 
20 identifier encoded in the image, the image file, or its metadata. For example, the session 
identifier may be a number or message embedded steganographically in the image or 
metadata associated with the image. 

Configuration via External Devices 
25 As noted in previous sections, the data to be associated with images may be provided 

by an external device. The external device may set operating parameters that control the 
operation of the camera, including how and what data it associates with images. The 
external device may also set the data types as well as the actual data to be associated with 
images in the camera. 
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In some applications, a computer can be programmed to perform this function 
automatically for one or more cameras. For example, a server may be programmed to 
broadcast operating parameters and data to several cameras in the field. 

In other applications, a user may enter the desired operating parameters and data to 
5 be associated with the images in the camera. In client-server scenario, for example, a user of 
a client device may submit desired operating parameters and data to server, which in turn, 
sends the information to the camera. In an alternative scenario, the user can enter this 
information via an application program executing on his PC, or PDA, and then instruct the 
application to download the information to the camera via a direct wire or wireless 
1 0 connection to the camera. 

Image Data Sources 

The data associated with images in the camera may come from a variety of sources. 
One source of data is the camera UI. Another source is the internal components and 

1 5 peripherals of the camera, such as a time clock, GPS device, camera components, camera 
software, etc. Yet another source is an external device. In this case, the camera will either 
have the data in its memory, after receiving it from the external source, or the camera will 
have a reference to data, which is stored on external device. As detailed further below, the 
reference may be an pointer, address or some other form of identifier that is encoded in the 

20 image. 

Storage of Data Types and Data Items Associated with Images 

A related design issue is where to store the data associated with an image at various 
stages of image processing. These stages include: 1) before configuring the camera; 2) after 
25 configuring, but before snapping the picture; 3) after snapping, but before transferring the 
associated image from the camera; and 4) after transferring the image from the camera. 
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Before Configuring the Camera 

Before configuring the camera, the data types and data items to be associated with 
images may be stored in the camera or in one or more external storage devices, depending 
on the application. 

5 The camera may maintain data types and items within the memory subsystem built 

into the camera (e.g., ROM or RAM) or within removable memory devices, such as a floppy 
disk, flash memory card etc. In some implementations, the camera maintains a set of data 
types that a user, application program or device may select to associate with images. A user 
can select from among these desired data types from the user interface of the camera or from 

1 0 an external device. 

In applications for external configuration, a user, device or application program may 
maintain this data in memory of a computer, such as personal computer, personal digital 
assistant, server on the Internet, etc. 
After Configuring the Camera 

15 After configuring the camera, selected data types and items to be associated with 

images may be stored in camera memory, or in external storage. In either case, the camera 
maintains an association between the images to be captured and the data to be associated 
with those images. In cases where the image metadata is stored externally, the camera 
maintains a reference to the external data, such as an identifier number, pointer, or address. 

20 After Capturing A Target Image 

After capturing a target image in the camera, the camera may store data items 
associated with the image in the image itself through steganographic encoding, in the image 
file, outside the image file but within camera memory, and in external storage. The "target" 
image refers to the image that is associated with selected data types and data items. In the 

25 last two cases, the camera maintains a reference between the target image and the associated 
data. The reference may be encoded in the image steganographically or in the image file. 
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After Transferring the Image from the Camera 

After transferring the image from the camera, the options for storing the data 
associated with the target image remain the same, yet the camera or some external device or 
process may transfer the data from one location to another. For example, an external 

5 steganographic encoder can embed some of the data in the image. In addition, an external 
process or device can transfer data to or from the image metadata in the image file. Image 
data stored externally may be transferred from one database, e.g., a configuration database, 
to another database, e.g., an image library or metadata database. For example, upon 
uploading an image to an image database, the uploading process can signal the configuration 

1 0 database to transfer image related data to the image or metadata database. 

Associating Image Data with Images 

The previous sections highlighted several methods for associating data with images. 
These methods include: steganographic encoding data in the image, storing the data within 
1 5 the image file (outside the image), storing the data outside the file and associating this 
external data with a reference stored in the file or encoded in the image. The following 
sections describe examples of these methods in more detail. 

Embedding Auxiliary Data within an Image Using Steganogra phic Methods 
The art of hiding auxiliary data into still and moving images is relatively advanced. 
20 Most techniques make slight changes to the image - e.g., by changing data representing the 
image, whether in the form of DCT coefficients, wavelet coefficients, pixel values, or other 
interchangeable representation - to thereby encode the auxiliary information. The changes 
are so slight as to be essentially imperceptible to the human visual system. Exemplary 
watermarking techniques are shown in 5,841,886 to Rhoads and U.S. Patent 5,915,027 to 
25 Cox. 

For additional information about a digital watermarking form of steganographic 

embedding and reading, see co-pending US Patent Application No. (Attorney 

Docket 60101), entitled WATERMARK EMBEDDER AND READER filed February 14, 
2000 by Rhoads et al., which is hereby incorporated by reference. The above references 
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describe how to embed auxiliary information steganographically in various media objects, 
including still image, video and audio signals. 

The steganographic encoder may be located within the camera or in external process 
or device. In either case, the encoder can be implemented in software, hardware or a 
5 combination of both. 

An encoder implementation in the camera operates on images stored in the camera's 
memory system. For each image, the encoder converts auxiliary data to be embedded in the 
image into watermark signal and combines the watermark signal with the image. This 
auxiliary data may include one or more references, a machine instruction or set of 

10 instructions, and other data items about the image. A reference is a data item, such as 
identifier number, address or pointer. It may be used to identify the image and reference 
additional data related to it. 

The camera-based encoder obtains auxiliary data to be embedded in an image from 
memory, or some other device in the camera. Recall that during configuration of the 

1 5 metadata, the user or some external device or process specifies the data types and data items 
to be associated with images. At encoding time, the encoder retrieves data designated for 
embedding in the image, converts it into a watermark signal and combines it with the 
associated image. To reduce the amount of data embedded in the image, the encoder may 
embed one or more references that identify data associated with the image that is stored 

20 outside the image. 

There are many places to implement the encoder outside the camera. In one 
implementation, the encoder is implemented as part of a process for uploading the image 
from the camera. In this implementation, the data for embedding in the image may come 
from the camera, an external device, or a combination of both. During the configuration 

25 stage, a configuration process specifies that data to be associated with image, including any 
data to be embedded in the image. As explained above, this data may be stored on the 
camera or an external device. The uploading process transfers the images from the camera, 
along with any data associated with the images. 

As part of the uploading process, an encoder then encodes auxiliary data designated 

30 for embedding in the image into the image. As in the camera-based process, this auxiliary 



JRM.dks 60100 pa 2/17/00 



-16- 



EXPRESS MAIL EL525676467US 



data may include one or more references that associates the image with other auxiliary data 
stored outside the image. This type of reference avoids the need to transfer metadata into 
the camera. Rather than transfer such data into the camera, metadata may be configured and 
maintained on an external device, and referenced to the image via a reference encoded into a 
watermark in the image. Conversely, the uploading process may move data associated with 
an image from the camera, store it in a metadata database, and reference the entry in the 
database via a reference encoded in the watermark. 

Embedding Auxiliary Data in the Image File 

The camera, or an external process, may also embed auxiliary data in the image file, 
but outside the image. To support this type of association, the image is stored in a file 
format that allows metadata to be stored in the file along with metadata. For example, the 
format may specify that the data is to be stored in the file header. This approach of 
associating data with an image has advantages, but is more susceptible to intentional or 
unintentional manipulation. 

In this case, a reference embedded in the image can be used to correlate the image 
with the correct metadata. This is particularly useful in applications where the metadata is 
placed in the image file by a process external to the camera. In these cases, it is possible that 
the metadata may be improperly assigned to a particular image. In addition, various image 
processing applications may remove or alter the image metadata stored in the image file. 
The data steganographically embedded in the image can be used to authenticate the metadata 
stored in the image file. One form of authentication is to match a reference number 
embedded in the image with a reference number in the image file metadata. More 
sophisticated forms of authentication may be implemented as well. 

For example, both the number in the file metadata and the watermark may be 
encrypted. Also, the number may be a digital signature derived from the image, such as by 
performing a hash function on the image sample values. The digital signature has the added 
advantage that it can be used to determine whether the image has been modified. For 
additional security, the digital signature can be encrypted, and then embedded in the 
watermark and placed in the image file metadata. 
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Maintaining Auxiliary Data Separate from the Associated Image 
As noted above, the auxiliary data associated with the image can be maintained 
separately from the image. A reference to this external, auxiliary data may be embedded in 
5 the image (e.g., in a watermark), in the image file metadata, or both. 



Data Types 

The types of data that can be associated with an image are numerous and vary with 
the application. The image data framework described above supports a variety of different 
1 0 data types, including: 

who took the picture (the photographer, 
who or what is the subject of the picture, 

where was the picture taken (e.g., entered by user, or provided by GPS device), 
when was the picture taken (e.g., date and time of day), 
15 • how was the picture taken, including camera specifications such as brand, model, 

manufacturer, and type of color converter, lens, sensor, flash, etc, 
picture specific device data, including shutter speed, flash used, camera settings, 
focus, distance of subject from camera, resolution of image, light levels, 
why was the picture taken? entertainment, legal record, medical record, real 
20 estate, business transaction record, etc. 

image type, 
copyright conditions, 
copyright owner and owner of copy, 
view data- the camera location and subject location, 
25 • comment- e.g., user comment 

picture identifiers, e.g., industry or application specific identifiers, 
collection data, including collection ID, collection caption, collection members, 
Image quality parameters, 
Color profile, 
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• Image integrity- Digital signature, encryption key (e.g., public encryption key), 

• Sound annotation, or reference to sound annotation, 

• Language specifier, for elements of the metadata, 

• view region- Advanced Photo System Print Aspect Ratio (PAR) selections of 
conventional, HDTV, and panoramic, 

• user address, e.g., email or physical address, 

• reference to derivative work- parent image and its derivatives, 

• reference to alternative images- e.g., alternative view, color space, resolution, etc. 

• region specification- describe a region in a picture, e.g., the pixels that represent 
an object in the picture, 

• compression format of image, compression format of image data (e.g., 
compression format of image metadata), 

• encryption type of the image, and the image data (encryption of image metadata), 

• references to other information (e.g., pointers, network links or addresses, phone 
numbers, etc.) 

• machine instructions or sets of instruction, e.g., instructions that control 
rendering of the image, that disable or enable certain types of editing operations, 
that control compression, decompression operations, etc. 



Metadata Applications that Exploit Steganographic Data 

Steganographic encoding of data enhances existing applications of image metadata 
and enables a variety of novel applications. The following sections describe some of these 
applications. 

Persistent Steganographic Data and Metadata Servers 

Steganographic encoding of auxiliary data in an image enables persistent linking of 
the image to its metadata. Metadata that is stored within an image file, yet outside the 
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image, is vulnerable to intentional and unintentional manipulation. Whenever an application 
or device processes the image, it may remove or alter the associated metadata. One solution 
is to store the metadata separately from the image. However, this approach requires that 
there be a reliable way to link the metadata to the image as it passes from one image 

5 processing application or device to the next. Steganographic encoding provides such a link 
between the image and its metadata. This link is referred to as a persistent staganographic 
link because it remains part of the image data through various image processing operations, 
including transfer to and from the analog domain (e.g., printing to image capture). 

Fig. 3 is a diagram illustrating a metadata server application that uses a persistent 

1 0 steganographic link to ensure that various devices and applications that process an image 
have access to its metadata. This diagram refers to "compliant" and "non-compliant" 
metadata applications. A compliant application refers to a device or software process that 
adheres to standard guidelines for maintaining image file metadata. A non-compliant 
application is a device or process that does not adhere to such guidelines, and may alter the 

1 5 metadata in unintended ways. 

Starting from the upper left of Fig. 3, a compliant application adds metadata to the 
image, and sends the metadata to a metadata server. The server stores the metadata along 
with a reference (e.g., unique number) to the image. This reference serves as a link to the 
image, and matches the reference steganographically embedded in the application. This 

20 example scenario assumes that the form of steganographic coding is a digital watermark. 
Either the compliant application, or some other previous process embeds the watermark, 
including the reference, in the image. 

The compliant application produces an image file that includes the digital 
watermarked image (DWM) and the metadata. One example of such an application is the 

25 application for uploading images from cameras or from PCs to a server on the Internet. This 
application embeds the watermark in the image and forwards the metadata to the metadata 
server. Note that the server application for uploading images and the metadata server 
database application may be implemented in the same server or a group of servers. 
Next, a non-compliant application gets this file, and removes or corrupts the 

30 metadata, leaving only the watermarked image. 
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Noticing that the image does not have metadata, the next compliant application sends 
a request to the metadata server for the image's metadata. The compliant application, 
equipped with a watermark detector, screens the image and extracts the reference to the 
image's metadata. It then forwards this reference to the metadata server, which returns the 
5 metadata. The format of the metadata may be based on XML or some other standard or 
custom data format. Preferably, the metadata format is sufficiently general to be compatible 
with many different devices and applications, but this not a requirement. The compliant 
application produces a new image file, including the watermarked image and its restored 
metadata. 

10 Metadata Router 

This concept can be extended beyond a single system by adding another component, 
the metadata router. Assume a collection of different systems, each with its own metadata 
server, and each tracking its own images. Fig. 4 depicts an example of three different 
systems, each having its own metadata server. Each of the metadata servers handles 

15 requests from imaging applications within its system. If an image from one system is 
acquired by another system, the new system will not have any associated metadata. 
Therefore, any application in the new system that tries to retrieve metadata from its metadata 
server will fail; it will find no information. If however, the new metadata server can 
determine which metadata server does have the information, it can request the image 

20 metadata from the other metadata server. 

The metadata router is the component that can help resolve which metadata server 
contains information about a particular image. The metadata router can operate either on a 
watermarked image, or a reference extracted from an image. In the former case, the 
requesting system has been unable to detect the metadata reference (e.g., it cannot detect a 

25 watermark), and thus, has forwarded the image to the router. The router then performs one 
or more watermark detection operations to extract the reference, and if successful, maps the 
extracted reference to the metadata server by using the extracted reference as a key to the 
metadata server in its database. 

In the latter case, the requesting system has extracted the metadata reference, but 

30 does not maintain the metadata for that reference. The metadata router maps the reference 
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to the appropriate metadata server by using the reference as a key to the metadata server in 
its database. It then redirects the request to the metadata server, which returns the requested 
metadata. The server having the metadata may return the metadata to the metadata server of 
the system from which the request originated and/or to the requesting application. This 
5 system may be implemented on a computer network, such as the Internet, using 

conventional networking protocols to communicate requests and return data over the 
network. 

The application may also make the request for metadata by requesting the metadata 
from the metadata router. The router, using the identifying information extracted from the 
1 0 watermark, redirects the request to the appropriate metadata server. The server maps the 
reference to the image's metadata and returns the requested metadata to the metadata server 
of the system from which the request originated and/or to the requesting application. 
Searching Metadata 

The techniques described above provide powerful applications for searching 
1 5 metadata that extend across different imaging applications and systems. For example, a user 
of an imaging application in one system can send a search request to the metadata server for 
that system. The search request can direct a search in one more fields of the metadata (e.g., 
time, place, subject, etc.) in the metadata database. Using the metadata router, the metadata 
server can extend the search across systems by forwarding the search request to a metadata 
20 router, which in turn, forwards the requests to other metadata servers. 

In addition, applications can support image based searching. For example, an end 
user might find an image of interest. The user then issues a request such as: "find other 
images like this one" (e.g., taken at a similar time, in a similar place, or having a similar 
subject). The client application or the metadata server extracts the reference from the image 
25 watermark. The metadata server then searches its database for images that are similar based 
on the criteria provided by the user, or based on metadata associated with the image. The 
metadata server can extend the search to other metadata servers by forwarding the request to 
other metadata servers via the metadata router. If the user finds an image that is not 
compatible with the system he is using, the same approach outlined above for finding the 
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image's metadata can be used to find the corresponding metadata server and initiate a 
database search for related images. 

Access to iust metadata (independent of the picture) 
5 The schemes outlined above facilitate access to metadata independent of the picture. 

By storing the metadata in a metadata database, applications can access, search and transfer 
the metadata without the images. This is particularly beneficial where bandwidth and 
storage is limited because the most common form of metadata, text, is usually smaller in size 
than the associated image. It requires less bandwidth and memory to transmit and store this 
1 0 text separately from its associated image. 

Metadata of an image may also be voluminous in some cases. For example, the 
metadata may include a sound or video annotation. It is advantageous to be able to store, 
access and search this type of metadata separately. 

Metadata editing (create, read, update, and delete) 
1 5 The metadata server scheme outlined above support editing of the metadata 

associated with an image. To edit an image's metadata, a user may access the metadata by 
extracting the persistent link to the metadata server and issuing a request to the metadata 
server. The metadata server implements a scheme for managing the rights to edit (e.g., 
create, read, update, and delete) the image's metadata. For example, the server may require 
20 the user to enter authentication information, such as user name and password. In response, 
the server determines the editing rights by looking them up in its database. Based on these 
rights, the user may be granted the authority to create, read, update, or delete the metadata. 

Metadata Persistence 

25 Persistent steganographic links to metadata ensure the persistence of an image's 

metadata through various operations, including file conversion (e.g., changing the file 
format of the image), file transmission (e.g., sending an image by email, or by wireless 
transmission), image compression, and image editing. The steganographic data is designed 
to stay in tact through various forms of image processing. 
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The persistence of the steganographic link also ensures metadata persistence of 
various types of data associated with the image, including copyright information. In 
addition, it supports adding multiple instances of copyright information, for works with 
multiple owners, and authors. The scheme for managing editing rights described above 
5 enables authors who create new works based on existing works to add copyright information 
to the image. The metadata server may be programmed to notify the copyright owner 
whenever another author wishes to modify an image. The right to modify a particular image 
may be pre-specified (e.g., a specified list of authors who have editing rights), or may be 
requested on a case by case basis. The metadata server may notify the copyright owner by 
10 email, for example, asking for authorization to grant editing rights to a particular author. If 
granted, the metadata server informs the new author of the terms. Then, if the new author 
excepts the terms by sending a return email or manifesting intent via an interactive web 
page, for example, the metadata server allows the new author to create a derivative work of 
the image. 

1 5 The derivative work inherits the metadata of the original work, including copyright 

information of the copyright owner in the original work. To associate the metadata for the 
new image, the metadata server provides a new reference to the new author's imaging 
application, which in turn, steganographically encodes the new reference in the derivative 
work. Additionally, the metadata server adds an entry in its database that associates the 

20 reference embedded in the new work with its corresponding metadata. This entry may also 
include a pointer to the database entry for the original work. This scheme for cross 
referencing earlier works enables the metadata server as well as system users and 
applications to track the history of an image through various types of editing. 

Steganographic linking of images to metadata allows the persistence of an image's 

25 metadata to be specified externally by a user, device or application. As noted above, the 
persistence of metadata is often vulnerable to intentional and unintentional manipulation. It 
is difficult for a user to specify which items of metadata are to persist with the image, and 
under which circumstances. 

The metadata server allows users, devices and application processes to specify which 

30 items of metadata are to remain persistent, through various forms of processing. The 
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metadata servers and routers, having the appropriate authority, may annotate an image's 
entry or entries in a metadata database to specify which data items will survive specified 
types of image processing, or image transfers. For example, if one system requests a 
metadata server to supply metadata to another system via the metadata router, the metadata 

5 server will only supply metadata that is designated to be persistent for the specified type of 
operation on the image. 

The steganographic link to metadata also supports the combination of metadata when 
two or more images are combined. As noted above, the metadata server may issue a new 
reference to be steganographically encoded in the new, combined image. The server may 

1 0 combine the metadata by combining the metadata for each component image into one 

metadata database entry, by inserting reference links in the database entry for the new image 
to the metadata entries for the component images, or some combination of these approaches. 
Metadata Format and Processing 

The metadata may be specified using the standard Extensible Markup Language, 
15 XML, or some other standard or custom format. The XML standard describes a class of 
data objects called XML documents and partially describes the behavior of computer 
programs which process them. XML is an application profile or restricted form of SGML, 
the Standard Generalized Markup Language [ISO 8879]. By construction, XML documents 
are conforming SGML documents. XML documents are made up of storage units called 
20 entities, which contain either parsed or unparsed data. Parsed data is made up of characters, 
some of which form character data, and some of which form markup. Markup encodes a 
description of the document's storage layout and logical structure. XML provides a 
mechanism to impose constraints on the storage layout and logical structure. 

A software module called an XML processor is used to read XML documents and 
25 provide access to their content and structure. In the implementations based on XML, the 
XML processor processes raw XML data on behalf of other application programs, such as 
an image processing application or a metadata server application. For more information on 
XML and XML processors, see the XML Standard document, Version 1.0, by the World 
Wide Web Consortium. 
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Metadata Security and Authentication 

The metadata associated with an image may be secured and authenticated at various 
stages of processing. These processing stages include: at or shortly after image capture in 
the camera, upon uploading the image from the camera, within the metadata server, and 

5 during transfers of the metadata from one device, application or system to another. One way 
to secure the metadata is to encrypt it. Another way is to restrict access to it via the 
metadata server such that only authenticated users, applications, and devices have access to 
it. Yet another way is to create a digital signature of the metadata, such as by performing a 
secret hash function on the data to derive a unique signature. This signature can be 

1 0 recomputed for suspect metadata and compared with a signature stored in the metadata 
database or elsewhere to determine whether the metadata has been modified. 

Other Applications 

With the growth in popularity of digital image capture devices, there is a growing 

15 demand for photo processing and storage services for digital images. Several companies 

now offer photo-processing services accessible via the Internet. These services create prints 
of digital photos sent via the Internet and mail the prints to the customer. A related service 
is providing an on-line database of images. Such services enable customers to maintain a 
collection of photos in an accessible database, e.g., an on-line database on the Internet. 

20 The metadata database applications described above provide an effective way to 

manage data associated with images. These applications are particularly applicable to the 
Internet, where the database information may be made available to a variety of users, 
imaging devices, application processes and systems. 

The image metadata may be maintained in a separate database or in a database 

25 integrated with an image database that stores the images themselves. The image database 
may contain a private collection of photos or collections of photos from unrelated users. 

A related application to maintaining image and metadata databases is tracking 
transactions involving the images or their metadata. The image or metadata database may 
keep a history or log file of the transactions associated with an image. For example, as users 

30 request processing services on photos, such as creating prints, putting the prints on objects 
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(e.g., shirts, cups, calendars, posters etc.), etc., a tracking application keeps a record of the 
transaction, listing attributes of the transaction like the date, vendor, service provided, and 
images subject to the service. The transaction history may then be linked with other 
metadata and the image itself by a reference, such as the steganographic link embedded in 
5 the image. There are a variety of other ways to link the transaction history with an image. 
For example, the transaction history may be stored in an entry in the metadata or image 
database associated with the image. 

These applications may also be implemented for other media signals, such as audio, 
video, graphical models, etc. as explained below. 

10 

Other Media Types 

The technology described above relating to persistent links in images also applies to 
other media signals, including audio, and video signals, and computer graphics models (e.g., 
two-dimensional, three-dimensional graphical models and animation). Video is just another 

1 5 form of an image signal, so the same technology applies to embedding, reading and using 
links and metadata steganographically encoded into video signals. Steganographic 
processes may also be applied to audio signals as described in the patent documents 
incorporated by reference in this document. As such, the same applications may be 
implemented for audio by embedding metadata and/or a persistent link to metadata in an 

20 audio signal. 

Much of the technology discussed above directly pertains to still image capture 
devices as well as video capture devices, so no further elaboration is required. Generally 
speaking, metadata or a persistent link to metadata may be steganographically encoded into 
a video signal at the time of capture or at some later time and location. A video capture 
25 device may be adapted to insert a steganographic link in one or more video frames as the 
video signal is being captured, or shortly thereafter, before the encoded signal is transferred 
from the device. Also, the steganographic link may be inserted in the video signal as it is 
being transferred to an external device, such as a computer either directly linked or 
indirectly linked through a communication network (e.g., the Internet). 
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Similarly, metadata or a persistent link to metadata may be steganographically 
encoded into an audio signal at the time of capture or at some later time and location. An 
audio capture device (e.g., an analog or digital audio recording device) may be adapted to 
insert a steganographic link in one or more audio segments as the audio signal is being 

5 captured, or shortly thereafter, before the encoded signal is transferred from the device. 

Also, the steganographic link may be inserted in the audio signal as it is being transferred to 
an external device, such as a computer either directly linked or indirectly linked through a 
communication network (e.g., the Internet). 

Once inserted in the media signal, a decoder can then extract the metadata or 

1 0 persistent link to metadata from the media signal Once a steganographic decoder extracts 
the link to metadata or metadata itself, the same applications described above for retrieving, 
processing and using metadata apply. For example, the metadata server and router systems 
may be adapted to store and control access to metadata for media signals other than images. 
The same metadata applications that exploit steganographic data for images apply to other 

1 5 media signals as well. 

Concluding Remarks 

Having described and illustrated the principles of the technology with reference to 
specific implementations, it will be recognized that the technology can be implemented in 

20 many other, different, forms. To provide a comprehensive disclosure without unduly 
lengthening the specification, applicants incorporate by reference the patents and patent 
applications referenced above. Additional information is attached in Appendix A, entitled 
"DIG35 Metadata and Digital Watermarking by Digimarc Corporation, which is also 
incorporated by reference. 

25 The particular combinations of elements and features in the above-detailed 

embodiments are exemplary only; the interchanging and substitution of these teachings with 
other teachings in this and the incorporated-by-reference patents/applications are also 
contemplated. 
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1 . An image capture device including: 
a sensor for capturing an image; and 

a steganographic encoder for encoding auxiliary data in the image. 

2. The image capture device of claim 1 wherein the auxiliary data describes 
attributes of the image. 

3. The image capture device of claim 1 wherein the auxiliary data includes a 
reference to auxiliary data stored outside the image. 

4. The image capture device of claim 1 wherein the auxiliary data includes 
authentication data for authenticated the image. 

5. The image capture device of claim 4 wherein the authentication data is derived 
from the image. 

6. The image capture device of claim 1 including an interface for receiving the 
auxiliary data from an external device. 

7. The image capture device of claim 1 including an interface for receiving an 
operating parameter specifying a type of auxiliary data to associate with the image. 

8. The image capture device of claim 1 including an interface for receiving session 
parameters that govern operation of the image capture device during a session. 

9. The image capture device of claim 8 wherein at least one of the session 
parameters instructs the image capture device to preclude a user from altering a camera 
setting during the session. 



JRM:dks 60100 pa 2/17/00 



-29- 



EXPRESS MAIL EL525676467US 



10. The image capture device of claim 8 wherein the session parameters specify 
auxiliary data to be associated with a picture during a session. 

1 1 . An image capture device including: 
5 a sensor for capturing an image; 

a processing unit for associating auxiliary data with the image; 
an interface for receiving session parameters that govern operation of the image 
capture device during a session. 

10 12. The image capture device of claim 1 1 wherein the session parameters specify 

auxiliary data to be associated with images captured by the sensor during a session. 

13. The image capture device of claim 1 1 wherein at least one of the session 
parameters instructs the image capture device to preclude a user from altering a camera 

15 setting during the session. 

14. The image capture device of claim 1 1 wherein a session identifier is 
steganographically encoded in image or metadata associated with the image in the session. 

20 15. A method for associating auxiliary data with an image, the method comprising: 

extracting a steganographic reference to auxiliary data in the image; 
querying a metadata database to request the auxiliary data associated with the image 
using the extracted reference to access the auxiliary data in the metadata database; and 
receiving the auxiliary data from the database. 

25 

16. The method of claim 15 including: 

using the auxiliary data received from the database to create an image file including 
the image and the auxiliary data. 
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17. The method of claim 15 including: 

sending authentication data to the metadata database to request access to the 
auxiliary data associated with the image. 

5 18. The method of claim 15 including: 

sending a request to the metadata database to edit the auxiliary data associated with 
the image. 

19. A computer readable medium having software for performing the method of 
10 claim 15. 

20. A method for associating auxiliary data with an image, the method comprising: 
maintaining a database of auxiliary data items associated with images, each item 

being associated with an image via a reference steganographically encoded in the image; 
1 5 from a requesting application, receiving a request for auxiliary data associated with 

an image, the request including a reference extracted from the image; 

using the reference to locate the auxiliary data associated with the image in the 
database; and 

returning the auxiliary data corresponding to the reference to the requesting 
20 application. 

21 . The method of claim 20 including: 

determining whether the requesting application has access rights to the requested 
auxiliary data. 



25 



22. The method of claim 20 including: 

determining whether the requesting application has editing rights for the requested 
auxiliary data. 
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23. A computer readable medium having software for performing the method of 
claim 20. 

24. A method for associating auxiliary data with an image, the method comprising: 
5 maintaining a database of steganographic links and metadata databases associated 

with the steganographic links; 

from a requesting application, receiving a steganographic link extracted from an 
image, the link referencing auxiliary data about the image; 

using the steganographic link to locate a metadata database that stores the auxiliary 
1 0 data associated with the image; and 

forwarding the steganographic link to the metadata database that stores the auxiliary 
data associated with the image. 

25. A computer readable medium having software for performing the method of 
15 claim 24. 

26. In an image capture system that includes a camera, a camera user interface, a 
distinct computer, and an interface for communicating between the camera and the 
computer, the camera including a timekeeping function, a method comprising: 

20 sending data from the computer to the camera to initiate a session, the data including 

date and time data; 

setting the timekeeping function in the camera to correspond to the date and time 
data received from the computer; and 

preventing the camera's timekeeping function from being changed, through the 
25 camera user interface, for the duration of the session. 

27. The method of claim 26 in which the session continues until photo data from the 
camera is downloaded to the computer. 
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28. In a method of associating auxiliary data with an image in an imaging system, 
the system including a camera, a computer, and an interface for communicating between the 
camera and the computer, an improvement comprising automatically steganographically 
encoding image data with digital watermark data upon transfer to the computer. 

5 

29. The method of claim 28 which includes associating metadata in the camera with 
an image captured in the camera, transferring said metadata to the computer with the image, 
and associating said metadata in the computer with the digital watermark. 

10 30. The method of claim 28 in which the digital watermark data permits detection of 

subsequent image alteration. 

3 1 . The method of claim 28 in which the encoding is performed by the computer. 

15 32. A method of operating an image capture system, the system including a camera 

and a distinct computer with a user interface, the method including providing to the camera 
from said computer at least one data item to be steganographically encoded in an image 
captured by the camera, wherein the user interface of the computer is utilized to specify the 
data item. 

20 

33. The method of claim 32 which includes providing a steganographic link to the 
camera from the computer. 

34. An image database accessible on the internet, permitting storage of images from 
25 plural unrelated consumers, characterized in that the database maintains - for each of several 

images - a history file detailing the vendors from whom a consumer has requested service 
relating to an image. 

35. A method of maintaining a transaction history for image processing transactions 
30 on images, the method comprising: 
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maintaining a transaction history of an image stored on a networked device; 

in response to receiving a transaction request to process an image stored on the 
networked device, updating the transaction history of the image by adding data about the 
transaction to the transaction history. 

5 

36. The method of claim 35 including: 

associating the transaction history with the image through a link between the image 
and the transaction history. 

10 37. The method of claim 36 wherein the link is a steganographic link embedded in 

the image. 
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ASSOCIATING DATA WITH IMAGES IN IMAGING SYSTEMS 



Abstract of the Disclosure 
A steganographic embedder associates data with an image by encoding the data, a 
link to the data, or a combination of both into the image. The embedder may be located in 
an image capture device or an external process or device. In one application, for example, 
an embedder process steganographically encodes data into an image as part of the process of 
uploading the image from the image capture device to external storage. 
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COMBINED DECLARATION AND POWER OF ATTORNEY 
FOR PATENT APPLICATION 

As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name, 

I believe I am the original, first and sole inventor (if only one name is listed below) or an 
original, first and joint inventor (if plural names are listed below) of the subject matter which is 
claimed' and for which a patent is sought on the invention entitled ASSOCIATING DATA WITH 
IMAGES IN IMAGING SYSTEMS, the specification of which 

[x] is attached hereto. 

[ ] was filed on as Application No. . 

[ ] was described and claimed in PCT International Application 

No. , filed on , and as amended 

under PCT Article 19 on (if applicable). 

[ ] and was amended on (if applicable). 

[ ] with amendments through (if applicable). 

I hereby state that I have reviewed and understand the contents of the above-identified 
specification, including the claims, as amended by any amendment referred to above. 

I acknowledge the duty to disclose information which is material to patentability as defined 
in Title 37, Code of Federal Regulations, § 1.56. If this is a continuation-in-part application filed 
under the conditions specified in 35 U.S.C. § 120 which discloses and claims subject matter in 
addition to that disclosed in the prior copending application, I further acknowledge the duty to 
disclose material information as defined in 37 CFR § 1.56 which occurred between the filing date 
of the prior application and the national or PCT international filing date of the continuation-in-part 
application. 

I hereby claim foreign priority benefits under Title 35, United States Code, § 119(a)-(d) of 
any foreign application(s) for patent or inventor's certificate or of any PCT International 
application(s) designating at least one country other than the United States of America listed below 
and have also identified below any foreign application(s) for patent or inventor's certificate or any 
PCT International application(s) designating at least one country other than the United States of 
America filed by me on the same subject matter having a filing date before that of the application(s) 
on which priority is claimed: 

Prior Foreign Application(s) Priority 

Claimed 

_[] [] 

(Number) (Country) {Day/Month/Year Filed) Yes No 
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I hereby claim the benefit under Title 35, United States Code, § 1 19(e) of any United States 
provisional application(s) listed below: 



Application Number Filing Date 

I hereby claim the benefit under Title 35, United States Code, § 120 of any United States 
appiication(s) or § 365(c) of any PCT International application(s) designating the United States, 
listed below and, insofar as the subject matter of each of the claims of this application is not 
disclosed in the prior United States or PCT International application in the manner provided by the 
first paragraph of Title 35, United States Code, § 1 12, I acknowledge the duty to disclose material 
information as defined in Title 37, Code of Federal Regulations, § 1.56(a) which occurred between 
the filing date of the prior application and the national or PCT International filing date of this 
application: 



(Application No.) (Filing Date) (Status: patented, 

Pending, abandoned) 

The undersigned hereby authorizes the U.S. attorney or agent named herein to accept and 

follow instructions from as to any action to be taken in the Patent and Trademark 

Office regarding this application without direct communication between the U.S. attorney or agent 
and the undersigned. In the event of a change in the persons from whom instructions may be 
taken, the U.S. attorney or agent named herein will be so notified by the undersigned. 

I hereby appoint the foilowing attorney(s) and/or agent(s) to prosecute this application, to 
file a corresponding international application, and to transact all business in the Patent and 
Trademark Office connected therewith: 



William Y. Conwell 
Joel R. Meyer 
Thomas M. Horgan 
Elmer Galbi 



Reg. No. 31,943 
Reg. No. 37,677 
Reg. No. 33,183 
Reg. No. 19,761 



Address all telephone calls to Joel R. Meyer at telephone number (503) 495-4644. 
Address all correspondence to: 



William Y. Conwell 

Digimarc Corporation 

19801 SW 72nd Avenue, Suite 250 

Tualatin, OR 97062 



I hereby declare that all statements made herein of my own knowledge are true and that ail 
statements made on information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false statements and the like so made are 
punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United States 
Code and that such willful false statements may jeopardize the validity of the application or any 
patent issued thereon. 
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Full Name of Sole or First Inventor: Bruce L. Davis 
Inventor's Signature 

Date 

Residence: Lake Oswego, Oregon 
Citizenship: USA 

Post Office Address: 15599 Village Drive, Lake Oswego, OR 97034 
Full Name of Second Joint Inventor: 

Inventor's Signature 

Date 

Residence: 

Citizenship: 

Post Office Address: 

Full Name of Third Joint Inventor: 

Inventor's Signature 

Date 

Residence: 

Citizenship: 

Post Office Address: 
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Metadata and Digital 
Watermarking 

By combining the DIG35 metadata standard, digital watermarking 
and asset management technologies, the flexibility, persistence and 
accuracy of image metadata can be enhanced 
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Low cost digital cameras and scanners, film to CD processing, and photo 
sharing web-sites are all helping fuel an explosive growth in digital imaging; a 
growth that both creates new opportunities for digital imaging and that brings 
with it new management problems. Whether you are a home enthusiast or a 
professional in the imaging industry, dealing with the sheer number of digital 
images is becoming a major problem. Fortunately, digital images, unlike their 
paper counterparts, can be part of the solution to the management problem. 

By combining the digital image with other data about that image -- i.e. image 
owner, location where the image was taken, and date and time when the image 
was taken -- we can develop applications and processes to manage the flood 
of digital images. Such additional information is referred to as metadata, or 
data about data. Digital images allow data to be embedded directly into an 
image, giving them a distinct advantage over paper images. 

Some tools such as Asset Management Systems and Workflow Applications 
try to address the management issues involved with digital imaging. These 
tools store metadata; in some form and in some location. But these systems 
are proprietary and do not enable the metadata gathered about an image in one 
device or application to be used in another device or application. In order to 
create devices, applications and processes that effectively and efficiently use 
metadata, they must all agree on the form of the metadata. 

The Digital Imaging Group's (DIG) DIG35 Initiative , building upon the XML 
(extensible Markup Language) standard, specifies the syntax and semantics 
of one such imaging metadata standard. This standard will enable different 
imaging devices, applications and processes to seamlessly work together. 
Additional work is being done by the <indecs> project and the DPI Initiative 
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to extend this interoperability beyond the imaging industry. By defining 
metadata frameworks and object identification, an infrastructure can be 
defined that will enable unprecedented levels of sharing and management of 
intellectual property. The remainder of this paper will focus on the DIG35 
Initiative and the image industry, but by extension, can be applied to other 
industries as well. 

While the DIG35 metadata standard provides a foundation for future 
development, real world issues cause problems with deployment today. Non- 
compliant image formats, imaging applications and devices, all contribute to 
weakening the reliability of imaging metadata. Either through unintentional, 
unavoidable or malicious usage, image metadata can be lost or altered. 
Additionally, there are situations where it may not be advantageous or proper 
to store the metadata with the image itself One solution to these situations is 
to store the metadata in a separate repository, a metadata server, and store a 
unique link within the image that references back to the separately stored 
metadata. 

The remaining problem with this solution is that the link within the image 
cannot itself be metadata. That is, the link needs to be part of the image, and 
not any extraneous information. (As we saw above, any extraneous 
information can be lost or modified.) Digital watermarking solves this 
problem by embedding a digital watermark directly into the image. Such a 
watermark uniquely identifies each image, and hence, provides the needed 
reference back to the metadata. 



The vision for the DIG35 Initiative Group is to "provide a standardized 
mechanism which allows end-users to see digital image use as being equally 
as easy, as convenient and as flexible as the traditional photographic methods 
while enabling additional benefits that are possible only with a digital format." 

The DIG35 Initiative Group has chosen to employ the W3C's Extensible 
Markup Language (XML) as the recommended mechanism to provide image 
metadata. The reasons for choosing XML include: 

• adoption in Internet enabled applications; 

• cross-platform nature; 

• extensibility; 

• device and operating system independence; and, 

• human and machine readable format. 



Confidential 



DRAFT -Page 2 



02/17/00 



DIG35 and Digital Watermarks 

Strengthening Digital image Metadata with Digital Watermarks 

The DIG35 Initiative Group then layers image metadata semantics on the 
XML underpinnings. This enables different image related applications (i.e. 
workflow, asset management, image capture) to effectively work together. 

For example, consider an insurance company that processes auto accident 
claims; claims that include images. These images may be acquired by digital 
cameras or scanners. As the claim (with images) goes through the claims 
approval process, these images may be modified by a number of different 
image editing applications on a number of different computing platforms. And 
throughout, the claims processing application will be manipulating the image 
metadata. In order to use the same metadata from the beginning of the claims 
approval process to the end, each device and application in the process must 
understand the format and meaning of the metadata. 

XML provides the means for understanding the format of the metadata, so it 
allows each application to read the metadata. But, it does not address the 
meaning, or semantics, of the metadata. For example, the digital camera may 
add date and time information about when the image was taken, the claims 
processing application may add date and time information about when the ^ 
claim was processed, and the image editing application may add date and time 
information about when the image was edited. Without a standard that 
specifies the meaning of each, information may be lost or corrupted. The 
DIG35 Initiative supplies the needed meaning to the metadata. 




As valuable as this image metadata is, it is also extremely vulnerable. First, 
DIG35 compliant image metadata relies on new and evolving image formats. 
Until such time as all imaging applications, processes and devices are 
upgraded to support the latest image formats in a sympathetic manner, DIG35 
metadata is vulnerable to accidental deletion or to unavoidable loss. Such 
vulnerabilities include: 

• Workflows where an image is converted to a non-DIG35 compliant image 
format and hence, loses the image metadata; 

• Imaging applications that don't preserve metadata during image 
manipulation (see Figure 1); and, 

• Converting a digital image to a non-digital format and subsequently, back 
to a digital format (e.g., printing out an image and then scanning it back 
in). 

• Preparing an image for the WWW where file size and download time need 
to be minimized. 
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A DJG35 Compliant application adds metadata to an image, only to have that metadata lost when the image is 

processed by a non-compliant application. 

Let's return to our example of the auto insurance company. Digital images are 
taken by the insurance agents and incorporated into the claim. A claims 
adjuster then processes this claim. Finally, a claims manager approves the 
claim. During this processing, the image may be moved from one computer 
platform to another (from a PC to a Unix Server to a Macintosh). For various 
reasons, the image may change from one format to another as it progresses 
from one computer to another (BMP to TIFF to JPEG). During each of these 
format changes, metadata may be lost if each image converter and each image 
format is not DIG35 compliant. The image may also be altered to facilitate its 
processing by adjusting its size or by converting to grayscale. During each of 
these alterations, metadata may again be lost if the image processing 
application is not DIG35 compliant. Finally, if the final claim is printed or 
faxed, all of the image metadata will be lost. 

The second image metadata vulnerability is malicious deletion or modification 
of the image metadata. By its nature, XML is stored as plain text. This makes 
it very easy to either change or delete the embedded image metadata. (Of 
particular note is the vulnerability of copyright information. With traditional 
film, one could assert a copyright by possession of the negative. However, 
with digital photography, there is no negative and hence, copyright 
enforcement requires new tools and techniques.) 

Digital watermarking helps address both of these types of vulnerabilities. By 
uniquely identifying each image, a link can be established and maintained 
between the image and its metadata. This enables tracking and asset 
management of images and interoperability between DIG35 devices and 
applications even when the embedded metadata has been lost or changed. 
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Since a digital watermark is embedded into the image data itself, the 
watermark cannot easily be changed or deleted without degrading the digital 
image. This enables a persistent link between image and metadata. By 
utilizing digital watermarks that uniquely identify images and storing DIG35 
image metadata in a separate metadata server, an image can always be 
identified and its image metadata retrieved even when using non-DIG35 
compliant applications or image formats. 
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Figure 2 

A DIG35 Compliant application adds metadata to an image and stores it with a metadata server. The metadata is 
lost when the image is processed by a non-compliant application, but is restored by a compliant application that 

retrieves the metadata from the metadata server. 

Continuing with our earlier example of auto insurance claims processing. 
When the image is first acquired, it is digitally watermarked and the metadata 
is copied to a metadata server. The image is originally saved in TIFF format 
on an IBM compatible PC. To support an Intranet based application, the 
image is subsequently converted to JPG and stored on a UNIX Web Server. In 
doing so, the embedded metadata is lost. Now, even though the image has 
been processed and the embedded metadata has been lost, users can still 
retrieve the associated metadata data from the metadata server by using the 
digital watermark found in the image itself. 

This concept can be extended beyond a single system by adding another 
component, the metadata router. The metadata router is used by an 
application to route a request for metadata to a remote metadata server. 
Assume a collection of different and distributed systems, each with its own 
metadata server. An application can request metadata about any image, 
without knowing which metadata server has that metadata. The application 
makes this request by requesting the data from the metadata router. The 
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router, using the identifying information in the watermark, can redirect the 
request to the appropriate metadata server. 




Metadata Server 



Figure 3: 

An Imaging Application requests metadata from the local Metadata Server, which forwards the request, via the 
Metadata Router, to an another Metadata Server. 

Finishing with our friendly auto insurance company, let's assume that there 
are a number of branch offices. Branch Office A begins a processes a claim, 
and storing all the image metadata on the local metadata server. Later, Branch 
Office B needs to do some additional work on the claim. During processing, 
they have occasion to need to access the metadata about a particular image. 
The application makes a request to the metadata router, where the router 
redirects the request to the metadata server at Branch Office A. The server 
there returns the requested metadata. 



While the rapid advances in digital imaging are opening up wonderful new 
possibilities for using digital images, it is imperative that we utilize new tools 
in the management and control of these digital images. The DIG35 Initiative 
provides a roadmap for future developments surrounding metadata enhanced 
digital imaging. 

To be useful, the metadata must be reliable. Today, with non-compliant 
applications and devices, the metadata cannot easily be integrated into 
existing processes, if at all. Additionally, due to file size or confidentiality 
concerns, it may not be desirable or proper to embed metadata directly into the 
image itself. By combining the DIG35 Metadata Standard, digital 
watermarking and metadata servers, it is possible to greatly increase the 
reliability and usability of image metadata. 
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Further, by utilizing a metadata router, the reach of the metadata sy stem is 
greatly increased. In addition to knowing metadata about images within its 
own system, the metadata server can discover information about images from 
other systems. 
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